fix(cdi): Include 32-bit libraries in discovery - #2035

Superseding and takeover of #1968.

Closes #1968

Reviewer: @henry118 @cdesiniotis

Continues #1968 by @elibosley, whose two commits are included here unmodified. The commits that follow address the review on this PR. Related to #563.

Background

ldcache.List() returns the 32-bit and 64-bit library sets separately, and the CDI lookup path discarded the 32-bit one. The default library locator also stopped at the first source that produced a match, so the linker cache could not contribute libraries from a compat32 directory once a library had been found in a predefined native path.

On a multilib host the generated spec therefore exposes the 64-bit driver stack while leaving a 32-bit application — a Steam or Proton title, a 32-bit CUDA or VDPAU consumer — without the matching vendor libraries, even though ELF32 driver libraries are installed and catalogued in the host's linker cache. Such applications fall back to software rendering or fail to start.

Where the gate is

The review asked for a flag to gate this, and for the 32-bit stack to be kept out of Alpine entirely. The gate is now in two parts, and the part that decides whether a container is affected is the container's own:

• The host spec carries the 32-bit stack whenever the host has it, with an opt-out for administrators who would rather it were not there at all.
• A container only sees it if it can use it. The update-ldcache hook decides this per container, at a point where both the ELF class of every injected library and the container's own userspace are directly observable.
• A musl container never sees it. i386 multiarch is not a supported concept in musl.

Discovery, on the host

The 32-bit set from ldcache.List() is no longer discarded, and the default library locator merges the unique matches of its sources instead of stopping at the first one that produces a match, so the linker cache can contribute a compat32 directory even when a predefined native path already matched. The native results keep their position at the front.

This is on by default rather than opt-in because:

• The 32-bit driver stack is already an opt-in at the packaging level. It is a separate, optional install on every distribution: the .run installer asks whether to install the 32-bit compatibility libraries, and Debian and Ubuntu ship them as a separate :i386 package. A host without them produces the same spec as today.
• Whether the libraries are wanted is a property of the workload, not of the host. A spec is generated once, by root or by a systemd unit, and a user who then needs the 32-bit stack cannot regenerate it. Gating discovery on the host fixes the answer for every container on the machine at install time, decided by someone who cannot know the question.
• What a container that does not want them pays is bounded and inert: the additional read-only bind mounts below, and nothing in its search path.

To leave them out of the spec entirely:

$ nvidia-ctk cdi generate --disable-compat32

# also honoured in /etc/nvidia-container-toolkit/nvidia-cdi-refresh.env,
# where it is documented as a commented example
NVIDIA_CTK_CDI_GENERATE_DISABLE_COMPAT32=true

[nvidia-container-runtime.modes.jit-cdi]
nvcdi-feature-flags = ["disable-compat32-libraries"]

and nvcdi.WithFeatureFlags(nvcdi.FeatureDisableCompat32Libraries) for API consumers such as the device plugin.

Exposure, in the container

internal/ldconfig classifies each injected directory by the ELF class of the libraries it contains (debug/elf) and adds a 32-bit directory to the container's search path only when:

• the container does not use musl, and
• it requested the compat32/all driver capability through NVIDIA_DRIVER_CAPABILITIES — the capability that drives --compat32 in nvidia-container-cli — or it ships a dynamic linker for 32-bit applications (/lib/ld-linux.so.2 or /lib32/ld-linux.so.2; /lib/ld-linux-armhf.so.3 or /lib/ld-linux.so.3 on arm64).

Otherwise the 32-bit directories are left out of the 00-nvcr-*.conf drop-in and of the musl .path file, and the dynamic linker never searches them. A stock ubuntu:24.04 or alpine container is therefore unaffected, and a multilib image — the one that has a reason to care — works with nothing set.

Notes on this:

• The capability is read from the container's OCI spec by the hook. Some configurations cannot read it (--userns=nomap, nvidia-cdi-hook fails when running a container with --userns nomap #648); a container whose capabilities cannot be determined is treated as not having requested them, and the linker check decides.
• Classification reads the ELF header rather than matching path names such as i386-linux-gnu or lib32, so it also holds for hosts that do not follow those conventions.
• The mounts themselves are the one thing this cannot gate: CDI container edits are static, so the libraries are bind-mounted into every container that requests the device, searchable or not. Gating those too would mean a separate device or class, which is a larger change than this one.

musl

/etc/ld-musl-<arch>.path carries no architecture information: musl's dynamic linker loads the first file matching the requested name and fails instead of continuing the search, so a wrong-arch match breaks the workload. There is no second .path file to route the 32-bit directories to, so they are never written to one. The hook additionally:

• prepends newly added driver directories ahead of the existing entries, so that an injected library wins a name lookup against a file of the same name that already sits in a searched directory. This is the same precedence the glibc path gives these directories through the 00-nvcr-*.conf drop-in;
• preserves musl's built-in /lib:/usr/local/lib:/usr/lib search path when it creates the file. Alpine ships no .path file, so the hook creates one, and creating it replaces the default search path. /usr/local/lib is silently dropped today; this is a pre-existing bug in the same function;
• splits the file on colons and newlines, exactly as musl's dynamic linker does, so the hook sees the same entries the linker will.

isMusl now also detects the musl dynamic linker, falling back to the /etc/alpine-release check it used before.

Verification

Bare-metal x86_64, Ubuntu 26.04, 2x Tesla P100-SXM2-16GB, driver 580.178.04, libnvidia-gl-580:i386 installed, Docker 29.7.2. Specs generated with nvidia-ctk cdi generate --mode=nvml into /etc/cdi under distinct vendors and requested with docker run --runtime=runc --device <vendor>/gpu=all.

Discovery. The generated spec grows from 58 to 83 hostPath entries: 25 ELF32 driver libraries under /usr/lib/i386-linux-gnu, their soname symlinks, and --folder /usr/lib/i386-linux-gnu{,/vdpau} for the update-ldcache hook.

The musl failure that motivates the container-side gate. With the 32-bit libraries in the spec and a hook that adds every injected directory to the .path file, in an Alpine container whose image ships one:

# ldd /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
Error loading shared library libnvidia-glsi.so.580.178.04: Exec format error (needed by /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0)
Error loading shared library libnvidia-tls.so.580.178.04: Exec format error (needed by /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0)
Error loading shared library libnvidia-glcore.so.580.178.04: Exec format error (needed by /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0)

With this PR, same image and spec:

# ldd /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
	libnvidia-glsi.so.580.178.04 => /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.580.178.04
	libnvidia-tls.so.580.178.04 => /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.580.178.04
	libnvidia-glcore.so.580.178.04 => /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.580.178.04

/etc/ld-musl-x86_64.path holds the native driver directories and no 32-bit entry. In the image above, which ships a .path file, the native driver directories are prepended and the existing entries keep their order.

32-bit workloads. In multilib images (ubuntu:26.04 and debian:trixie with libc6:i386, libvulkan1{,:i386}, libegl1{,:i386}, libx11-6{,:i386} and libxext6{,:i386}), a 32-bit process enumerates both P100s through Vulkan (vkCreateInstance, vkEnumeratePhysicalDevices, vkGetPhysicalDeviceProperties, served by the injected 32-bit libGLX_nvidia.so.0 ICD) and initialises the CUDA driver API (cuInit succeeds and reports both devices through the 32-bit libcuda.so.1). On main the same programs fail with VK_ERROR_INCOMPATIBLE_DRIVER and a failed dlopen, while their 64-bit builds pass either way. These images carry libc6:i386, so the hook exposes the 32-bit directories to them with nothing set. The ICD dlopens glvnd's libEGL.so.1 and the X11 client libraries of its own architecture, so the image has to provide those; without them it fails identically on main.

Second host, discovery and the gating decision. x86_64, Tesla V100-SXM2-32GB, driver 580.173.02, libnvidia-gl-580:i386 installed. nvidia-ctk cdi generate --mode=nvml produces 83 mounts where main produces 58, the 25 additional entries being the ELF32 driver libraries under /usr/lib/i386-linux-gnu{,/vdpau}, which are also passed to the update-ldcache hook as --folder arguments; --disable-compat32 produces a spec identical to the one generated on main. The hook's decision, run against real filesystems rather than fixtures: that host's own root, which carries /lib/ld-linux.so.2, is allowed with nothing set; an unpacked alpine-minirootfs-3.21.3 is skipped, with and without the compat32 capability; and the ELF class filter over the four driver directories keeps /usr/lib/x86_64-linux-gnu{,/vdpau} and drops /usr/lib/i386-linux-gnu{,/vdpau}.

Unit tests cover the locator merge and the ldcache split (from #1968), the discovery opt-out, the ELF class filter, the container-side gate in all of its cases, the .path file ordering and default-search-path preservation, musl detection, and the container environment lookup the capability is read from. Tests for the touched packages pass, and golangci-lint run ./..., go vet ./... and gofmt are clean.

Notes for reviewers

• NVIDIA_CTK_LIBCUDA_DIR becomes /usr/lib/x86_64-linux-gnu:/usr/lib/i386-linux-gnu on multilib hosts. The variable already carries a list, and the native directory stays first, but any consumer taking the last entry would now get the 32-bit one.
• Merging in NewLibraryLocator widens every lookup that uses it, not only 32-bit ones: the ldcache is now consulted even when a predefined search path already matched. Deduplication and native-first ordering keep version inference, which uses the first result, unchanged. --disable-compat32 restores the previous first-match behaviour along with the rest.
• The container-side gate lives in the hook, so a spec generated by this version and run against an older nvidia-cdi-hook would not be gated. The two ship in the same package.
• Hosts that pass explicit search paths — NixOS drives nvidia-ctk with --library-search-path — take the early return in NewLibraryLocator and are unaffected by the discovery change. Exposing the 32-bit stack there means passing a second search path for the driver's lib32 output; the hook change classifies those directories correctly, since it does not rely on path naming.
• Prepending in the musl .path file only applies to directories that are not searched already, so a driver directory that is also a system directory keeps its position.